{
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LASAZXP9yNdd"
      },
      "outputs": [],
      "source": [
        "# Copyright 2020 Google LLC\n",
        "#\n",
        "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "#     https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MGTixX2zyNdh"
      },
      "source": [
        "<table align=\"left\">\n",
        "    <td>\n",
        "        <a href=\"https://colab.research.google.com/github/GoogleCloudPlatform/ai-platform-samples/blob/master/notebooks/samples/optimizer/ai_platform_vizier_tuner.ipynb\">\n",
        "            <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\"> Run in Colab\n",
        "        </a>\n",
        "    </td>\n",
        "    <td>\n",
        "        <a href=\"https://github.com/GoogleCloudPlatform/ai-platform-samples/blob/master/notebooks/samples/optimizer/ai_platform_vizier_tuner.ipynb\">\n",
        "            <img src=\"https://cloud.google.com/ml-engine/images/github-logo-32px.png\" alt=\"GitHub logo\">View on GitHub\n",
        "        </a>\n",
        "     </td>\n",
        "</table>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ylcY4n8_yNdh"
      },
      "source": [
        "# Overview\n",
        "\n",
        "This tutorial demonstrates AI Platform's CloudTuner service.\n",
        "\n",
        "### Objective\n",
        "\n",
        "CloudTuner is implemented based upon the KerasTuner and uses AI Platform Vizier as an oracle to get suggested trials, run trials, etc. The usage of CloudTuner is the same as KerasTuner and additionally accept Vizier's `study_config` as an alternative input.\n",
        "\n",
        "### Costs\n",
        "\n",
        "This tutorial uses billable components of Google Cloud:\n",
        "\n",
        "* AI Platform Training\n",
        "* Cloud Storage\n",
        "\n",
        "Learn about [AI Platform Training\n",
        "pricing](https://cloud.google.com/ai-platform/training/pricing) and [Cloud Storage\n",
        "pricing](https://cloud.google.com/storage/pricing), and use the [Pricing\n",
        "Calculator](https://cloud.google.com/products/calculator/)\n",
        "to generate a cost estimate based on your projected usage.\n",
        "\n",
        "### PIP install packages and dependencies\n",
        "\n",
        "Install additional dependencies not installed in the notebook environment.\n",
        "\n",
        "- Use the latest major GA version of the framework.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sGY1bL9HyNdi"
      },
      "outputs": [],
      "source": [
        "! pip install google-cloud\n",
        "! pip install google-cloud-storage\n",
        "! pip install requests\n",
        "! pip install tensorflow_datasets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a1wR3ULtyNdq"
      },
      "source": [
        "### Set up your Google Cloud project\n",
        "\n",
        "**The following steps are required, regardless of your notebook environment.**\n",
        "\n",
        "1. [Select or create a Google Cloud project.](https://console.cloud.google.com/cloud-resource-manager) When you first create an account, you get a $300 free credit towards your compute/storage costs.\n",
        "\n",
        "2. [Make sure that billing is enabled for your project.](https://cloud.google.com/billing/docs/how-to/modify-project)\n",
        "\n",
        "3. [Enable the AI Platform APIs](https://console.cloud.google.com/flows/enableapi?apiid=ml.googleapis.com)\n",
        "\n",
        "4. If running locally on your own machine, you will need to install the [Google Cloud SDK](https://cloud.google.com/sdk).\n",
        "\n",
        "**Note**: Jupyter runs lines prefixed with `!` as shell commands, and it interpolates Python variables prefixed with `$` into these commands."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M5fbQ27KyNds"
      },
      "source": [
        "### Authenticate your Google Cloud account\n",
        "\n",
        "**If you are using [AI Platform Notebooks](https://cloud.google.com/ai-platform/notebooks/docs/)**, your environment is already\n",
        "authenticated. Skip these steps."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3v4NtseqyNdt"
      },
      "outputs": [],
      "source": [
        "import sys\n",
        "\n",
        "# If you are running this notebook in Colab, run this cell and follow the\n",
        "# instructions to authenticate your Google Cloud account. This provides access\n",
        "# to your Cloud Storage bucket and lets you submit training jobs and prediction\n",
        "# requests.\n",
        "\n",
        "if 'google.colab' in sys.modules:\n",
        "    from google.colab import auth as google_auth\n",
        "    google_auth.authenticate_user()\n",
        "\n",
        "# If you are running this tutorial in a notebook locally, replace the string\n",
        "# below with the path to your service account key and run this cell to\n",
        "# authenticate your Google Cloud account.\n",
        "else:\n",
        "    %env GOOGLE_APPLICATION_CREDENTIALS your_path_to_credentials.json\n",
        "\n",
        "# Log in to your account on Google Cloud\n",
        "! gcloud auth application-default login\n",
        "! gcloud auth login"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GeTcfKkv5NbV"
      },
      "source": [
        "#### Install CloudTuner\n",
        "\n",
        "Download and install CloudTuner from tensorflow-cloud."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "34uz7lC_5ODj"
      },
      "outputs": [],
      "source": [
        "! pip install tensorflow-cloud"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hKxM_8iQ527Q"
      },
      "source": [
        "#### Restart the Kernel\n",
        "\n",
        "We will automatically restart your kernel so the notebook has access to the packages you installed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-pVrJUUp53oC"
      },
      "outputs": [],
      "source": [
        "# Restart the kernel after pip installs\n",
        "import IPython\n",
        "app = IPython.Application.instance()\n",
        "app.kernel.do_shutdown(True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "79bOVMH4yNdw"
      },
      "source": [
        "### Import libraries and define constants"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KE24lBqTyNdw"
      },
      "outputs": [],
      "source": [
        "from tensorflow_cloud import CloudTuner\n",
        "import kerastuner\n",
        "\n",
        "REGION = 'us-central1'\n",
        "PROJECT_ID = '[your-project-id]' #@param {type:\"string\"}\n",
        "! gcloud config set project $PROJECT_ID"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xtX0LA3wyNd2"
      },
      "source": [
        "## Tutorial\n",
        "\n",
        "### Prepare Data\n",
        "\n",
        "For this tutorial, we will use a subset (10000 examples) from the MNIST dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GJSDGxmTyNd2"
      },
      "outputs": [],
      "source": [
        "from tensorflow.keras.datasets import mnist\n",
        "(x, y), (val_x, val_y) = mnist.load_data()\n",
        "x = x.astype('float32') / 255.\n",
        "val_x = val_x.astype('float32') / 255.\n",
        "\n",
        "x = x[:10000]\n",
        "y = y[:10000]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tgleaEx_yNd4"
      },
      "source": [
        "### Define model building function\n",
        "\n",
        "Next, we will define the hyperparameter model building function like one does for KerasTuner, where the following are tunable:\n",
        "\n",
        "    - number of layers\n",
        "    - the learning rate\n",
        "\n",
        "*Note that CloudTuner does not support adding hyperparameters in the model building function. Instead, the search space is configured by passing a `hyperparameters` argument when instantiating (constructing) the tuner.*"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "g5R1D-j7yNd4"
      },
      "outputs": [],
      "source": [
        "from tensorflow.keras import Sequential\n",
        "from tensorflow.keras.layers import Flatten, Dense\n",
        "from tensorflow.keras.optimizers import Adam\n",
        "\n",
        "\n",
        "def build_model(hp):\n",
        "    model = Sequential()\n",
        "    model.add(Flatten(input_shape=(28, 28)))\n",
        "\n",
        "    # the number of layers is tunable\n",
        "    for _ in range(hp.get('num_layers')):\n",
        "        model.add(Dense(units=64, activation='relu'))\n",
        "    model.add(Dense(10, activation='softmax'))\n",
        "\n",
        "    # the learning rate is tunable\n",
        "    model.compile(\n",
        "        optimizer=Adam(lr=hp.get('learning_rate')),\n",
        "        loss='sparse_categorical_crossentropy',\n",
        "        metrics=['accuracy'])\n",
        "    return model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6743HmI0yNd6"
      },
      "source": [
        "### Instantiate CloudTuner\n",
        "\n",
        "Next, we instantiate an instance of the CloudTuner. We will define our tuning hyperparameters and pass them into the constructor as the parameter `hyperparameters`.\n",
        "\n",
        "We also set the objective (`'accuracy'`) to measure the performance of each trial, and we shall keep the number of trials small (5) for the purpose of this demonstration."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NZ_HB4j_yNd6"
      },
      "outputs": [],
      "source": [
        "# Configure the search space\n",
        "HPS = kerastuner.engine.hyperparameters.HyperParameters()\n",
        "HPS.Float('learning_rate', min_value=1e-4, max_value=1e-2, sampling='log')\n",
        "HPS.Int('num_layers', 2, 10)\n",
        "\n",
        "tuner = CloudTuner(\n",
        "    build_model,\n",
        "    project_id=PROJECT_ID,\n",
        "    region=REGION,\n",
        "    objective='accuracy',\n",
        "    hyperparameters=HPS,\n",
        "    max_trials=5,\n",
        "    directory='tmp_dir/1')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HFrnrwTpyNd9"
      },
      "source": [
        "Let's use the `search_space_summary()` method to display what the search space for this optimization study looks like."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GC0KRPXJyNd-"
      },
      "outputs": [],
      "source": [
        "tuner.search_space_summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hit4PNwiyNd_"
      },
      "source": [
        "### Search\n",
        "\n",
        "Let's now execute our search for this optimization study with the `search()` method. This method takes the same parameters as the `fit()` method in TF.keras API model instance."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FDBYRAb5yNeA"
      },
      "outputs": [],
      "source": [
        "tuner.search(x=x, y=y, epochs=10, validation_data=(val_x, val_y))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2wFed7rzyNeC"
      },
      "source": [
        "### Results\n",
        "\n",
        "Next, we use the `results_summary()` method to get a summary of the trials that were tried in this optimization study."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YcBd2uWCyNeC"
      },
      "outputs": [],
      "source": [
        "tuner.results_summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JAMb9rumyNeE"
      },
      "source": [
        "### Get the Best Model\n",
        "\n",
        "Now, let's get the best model from the study using the `get_best_models()` method. The parameter `num` specifies the topmost number of models. In our case, we set it to `1` for the best overall model. The method returns a list (of models), so we use index of `0` to get the model out of the list."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "as1kXWuayNeE"
      },
      "outputs": [],
      "source": [
        "model = tuner.get_best_models(num_models=1)[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Y86ESL2z7J4b"
      },
      "outputs": [],
      "source": [
        "print(model)\n",
        "print(model.weights)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f9pXZgwVyNeG"
      },
      "source": [
        "## Tutorial: Using an input pipeline with datasets\n",
        "\n",
        "In this example we will build training pipeline that uses [`tf.data.datasets`](https://www.tensorflow.org/datasets/overview) for training the model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xECKBXD1XflX"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "import tensorflow_datasets as tfds"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l4JsqEZeXfla"
      },
      "source": [
        "### Load MNIST Data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mvjf5xihXfla"
      },
      "outputs": [],
      "source": [
        "(ds_train, ds_test), ds_info = tfds.load(\n",
        "    'mnist',\n",
        "    split=['train', 'test'],\n",
        "    shuffle_files=True,\n",
        "    as_supervised=True,\n",
        "    with_info=True,\n",
        ")\n",
        "\n",
        "# tfds.load introduces a new logger which results in duplicate log messages.\n",
        "# To mitigate this issue following removes Jupyter notebook root logger handler. More details @\n",
        "# https://stackoverflow.com/questions/6729268/log-messages-appearing-twice-with-python-logging\n",
        "\n",
        "import logging\n",
        "logger = logging.getLogger()\n",
        "logger.handlers = []"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8PVU3SDKXfld"
      },
      "source": [
        "### Build training pipeline\n",
        "Build a training and evaluation pipeline using `ds.map`, `ds.cache`, `ds.shuffle`, `ds.batch`, and `ds.prefetch`. For more details on building high performance pipelines refer to [data performance](https://www.tensorflow.org/guide/data_performance)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Xg2lOPanXfle"
      },
      "outputs": [],
      "source": [
        "def normalize_img(image, label):\n",
        "    \"\"\"Normalizes images: `uint8` -> `float32`.\"\"\"\n",
        "    return tf.cast(image, tf.float32) / 255., label\n",
        "\n",
        "\n",
        "ds_train = ds_train.map(\n",
        "    normalize_img, num_parallel_calls=tf.data.experimental.AUTOTUNE)\n",
        "ds_train = ds_train.cache()\n",
        "ds_train = ds_train.shuffle(ds_info.splits['train'].num_examples)\n",
        "ds_train = ds_train.batch(128)\n",
        "ds_train = ds_train.prefetch(tf.data.experimental.AUTOTUNE)\n",
        "\n",
        "ds_test = ds_test.map(\n",
        "    normalize_img, num_parallel_calls=tf.data.experimental.AUTOTUNE)\n",
        "ds_test = ds_test.batch(128)\n",
        "ds_test = ds_test.cache()\n",
        "ds_test = ds_test.prefetch(tf.data.experimental.AUTOTUNE)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J14hQyl7Xflf"
      },
      "source": [
        "### Create and train the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6YUgiFN_Xflg"
      },
      "outputs": [],
      "source": [
        "def build_pipeline_model(hp):\n",
        "    model = Sequential()\n",
        "    model.add(Flatten(input_shape=(28, 28, 1)))\n",
        "\n",
        "    # the number of layers is tunable\n",
        "    for _ in range(hp.get('num_layers')):\n",
        "        model.add(Dense(units=64, activation='relu'))\n",
        "    model.add(Dense(10, activation='softmax'))\n",
        "\n",
        "    # the learning rate is tunable\n",
        "    model.compile(\n",
        "        optimizer=Adam(lr=hp.get('learning_rate')),\n",
        "        loss='sparse_categorical_crossentropy',\n",
        "        metrics=['accuracy'])\n",
        "    return model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Qxj2lZWeXfli"
      },
      "outputs": [],
      "source": [
        "# Configure the search space\n",
        "pipeline_HPS = kerastuner.engine.hyperparameters.HyperParameters()\n",
        "pipeline_HPS.Float('learning_rate', min_value=1e-4, max_value=1e-2, sampling='log')\n",
        "pipeline_HPS.Int('num_layers', 2, 10)\n",
        "\n",
        "pipeline_tuner = CloudTuner(\n",
        "    build_pipeline_model,\n",
        "    project_id=PROJECT_ID,\n",
        "    region=REGION,\n",
        "    objective='accuracy',\n",
        "    hyperparameters=pipeline_HPS,\n",
        "    max_trials=5,\n",
        "    directory='tmp_dir/2')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cuNCTP91Xflk"
      },
      "outputs": [],
      "source": [
        "pipeline_tuner.search(x=ds_train, epochs=10, validation_data=ds_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "xxQ5HPf_Xflm"
      },
      "outputs": [],
      "source": [
        "pipeline_tuner.results_summary()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bjK1gXdBXflp"
      },
      "outputs": [],
      "source": [
        "pipeline_model = pipeline_tuner.get_best_models(num_models=1)[0]\n",
        "print(pipeline_model)\n",
        "print(pipeline_model.weights)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1OhLYJBYXflr"
      },
      "source": [
        "## Tutorial: Using a Study Configuration\n",
        "\n",
        "Now, let's repeat this study but this time the search space is passed in as a Vizier `study_config`.\n",
        "\n",
        "### Create the Study Configuration\n",
        "\n",
        "Let's start by constructing the study config for optimizing the accuracy of the model with the hyperparameters number of layers and learning rate, just as we did before."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oy3WVQfwyNeH"
      },
      "outputs": [],
      "source": [
        "# Configure the search space\n",
        "STUDY_CONFIG = {\n",
        "    'algorithm': 'ALGORITHM_UNSPECIFIED',\n",
        "    'metrics': [{\n",
        "        'goal': 'MAXIMIZE',\n",
        "        'metric': 'accuracy'\n",
        "    }],\n",
        "    'parameters': [{\n",
        "        'discrete_value_spec': {\n",
        "            'values': [0.0001, 0.001, 0.01]\n",
        "        },\n",
        "        'parameter': 'learning_rate',\n",
        "        'type': 'DISCRETE'\n",
        "    }, {\n",
        "        'integer_value_spec': {\n",
        "            'max_value': 10,\n",
        "            'min_value': 2\n",
        "        },\n",
        "        'parameter': 'num_layers',\n",
        "        'type': 'INTEGER'\n",
        "    }, {\n",
        "        'discrete_value_spec': {\n",
        "            'values': [32, 64, 96, 128]\n",
        "        },\n",
        "        'parameter': 'units',\n",
        "        'type': 'DISCRETE'\n",
        "    }],\n",
        "    'automatedStoppingConfig': {\n",
        "        'decayCurveStoppingConfig': {\n",
        "            'useElapsedTime': True\n",
        "        }\n",
        "    }\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d-U1lUwAyNeJ"
      },
      "source": [
        "### Instantiate CloudTuner\n",
        "\n",
        "Next, we instantiate an instance of the CloudTuner. In this instantiation, we replace the `hyperparameters` and `objective` parameters with the `study_config` parameter."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Or_5NloLyNeJ"
      },
      "outputs": [],
      "source": [
        "tuner = CloudTuner(\n",
        "    build_model,\n",
        "    project_id=PROJECT_ID,\n",
        "    region=REGION,\n",
        "    study_config=STUDY_CONFIG,\n",
        "    max_trials=10,\n",
        "    directory='tmp_dir/3')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OuyRbyyUyNeL"
      },
      "source": [
        "Let's use the `search_space_summary()` method to display what the search space for this optimization study looks like."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oHcplJLNyNeM"
      },
      "outputs": [],
      "source": [
        "tuner.search_space_summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_JCa2YU1yNeN"
      },
      "source": [
        "### Search\n",
        "\n",
        "Let's now execute our search for this optimization study with the `search()` method."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OfQFb76CyNeO"
      },
      "outputs": [],
      "source": [
        "tuner.search(x=x, y=y, epochs=5, steps_per_epoch=2000, validation_steps=1000, validation_data=(val_x, val_y))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zPckxD79yNeP"
      },
      "source": [
        "### Results\n",
        "\n",
        "Now let's use the `results_summary()` method to get a summary of the trials that were tried in this optimization study."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LK7svO7XyNeQ"
      },
      "outputs": [],
      "source": [
        "tuner.results_summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ek73BjAGGURj"
      },
      "source": [
        "## Tutorial: Distributed Tuning\n",
        "Let's run multiple tuning loops concurrently using multiple threads. To run distributed tuning, multiple tuners should share the same `study_id`, but different `tuner_id`s."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jnO_SFGmG3pE"
      },
      "outputs": [],
      "source": [
        "from multiprocessing.dummy import Pool\n",
        "# If you are running this tutorial in a notebook locally, you may run multiple\n",
        "# tuning loops concurrently using multi-processes instead of multi-threads.\n",
        "# from multiprocessing import Pool\n",
        "\n",
        "import time\n",
        "import datetime\n",
        "\n",
        "STUDY_ID = 'Tuner_study_{}'.format(\n",
        "    datetime.datetime.now().strftime('%Y%m%d_%H%M%S'))\n",
        "\n",
        "\n",
        "def single_tuner(tuner_id):\n",
        "    \"\"\"Instantiate a `CloudTuner` and set up its `tuner_id`.\n",
        "\n",
        "    Args:\n",
        "        tuner_id: Integer.\n",
        "    Returns:\n",
        "        A CloudTuner.\n",
        "    \"\"\"\n",
        "    tuner = CloudTuner(\n",
        "        build_model,\n",
        "        project_id=PROJECT_ID,\n",
        "        region=REGION,\n",
        "        objective='accuracy',\n",
        "        hyperparameters=HPS,\n",
        "        max_trials=18,\n",
        "        study_id=STUDY_ID,\n",
        "        directory=('tmp_dir/cloud/%s' % (STUDY_ID)))\n",
        "    tuner.tuner_id = str(tuner_id)\n",
        "    return tuner\n",
        "\n",
        "\n",
        "def search_fn(tuner):\n",
        "    # Start searching from different time points for each worker to avoid `model.build` collision.\n",
        "    time.sleep(int(tuner.tuner_id) * 2)\n",
        "    tuner.search(x=x, y=y, epochs=5, validation_data=(val_x, val_y), verbose=0)\n",
        "    return tuner\n",
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iV2Pi-GWIRMV"
      },
      "source": [
        "### Search\n",
        "\n",
        "Let's now execute multiple search loops in parallel for this study with the `search()` method."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5N4Ev5TxH2AW"
      },
      "outputs": [],
      "source": [
        "# Number of search loops we would like to run in parallel\n",
        "num_parallel_trials = 4\n",
        "tuners = [single_tuner(i) for i in range(num_parallel_trials)]\n",
        "p = Pool(processes=num_parallel_trials)\n",
        "result = p.map(search_fn, tuners)\n",
        "p.close()\n",
        "p.join()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9c9ErYBxJJ1M"
      },
      "source": [
        "### Results\n",
        "\n",
        "Now let's use the `results_summary()` method to get a summary of the trials (from all the search loops) that were tried in this optimization study."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2G7KKDbGJJ_U"
      },
      "outputs": [],
      "source": [
        "result[0].results_summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RERy47XnyNeR"
      },
      "source": [
        "# Cleaning up\n",
        "\n",
        "To clean up all Google Cloud resources used in this project, you can [delete the Google Cloud\n",
        "project](https://cloud.google.com/resource-manager/docs/creating-managing-projects#shutting_down_projects) you used for the tutorial."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "ai_platform_vizier_tuner.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
